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METHOD AND APPARATUS FOR RECOGNIZING A STRUCTURE 
TO BE APPLIED ONTO A SUBSTRATE WITH MULTIPLE 
CAMERAS , AND DEVICE THEREFORE 

[0001] The present application relates to and claims priority from European Application 
Serial No. PCT/EP2004/0 14698 filed December 23, 2004. titled "METHOD FOR 
RECOGNIZING A STRUCTURE TO BE APPLIED TO A SUBSTRATE. WITH THE AID OF 
SEVERAL CAMERAS AND EVICE THEREFORE", the complete subject matter of which is 
hereby expressly incorporated in its entirety. 

[0002] The present invention relates to a method for recognizing a structure to be applied 
onto a substrate with at least one or more cameras in accordanc e with the g e neric part of claim , 
and a corresponding apparatus therefore . 

[0003] For recognizing a structure to be applied onto a substrate it has been customary to 
carry out optical measurements . Frequently , wh e r e by fr e qu e ntly various systems for fully 
automatic testing of the structure, including adhesive and sealing agent lines, are used^. for this 
purpos e . For this purpose, multiple video cameras are directed at the structure to be recognized. 
In , wh e r e by, in addition, an illumination module serving to generate a contrast-rich camera 
image is required. 

[0004] In order to be able to monitor an adhesive line and/or adhesive trail while the 
structure it is being applied, it is necessary to teach-in a reference adhesive trailj^-i.e.^ to have the 
camera or cameras scan the reference adhesive tail)? in order to calculate ther e from 
corresponding parameters on which the assessment of the applied adhesive trails are is bas e d 
subsequently based . 

[0005] Conventionally there Until now, it was a necessity n e c e ssary in the teaching-in of a 
reference adhesive trail for each camera to individually scan the adhesive trail in order to obtain 
camera images for all positions. When using For th e u se of three cameras, this m e ans that the 
reference adhesive trail had to be scanned thrice in sequence and the three different sequences of 
images of the three cameras had to be assigned. A disadvantage with this method is that the This 
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is disadvantageous in that it re s ults in the parameterization of the reference adhesive trail being 
cumbersome and time tkfle consuming and can lead to high inaccuracy. 

[0006] Moreover, there is a need for a method to recognize for r e cognizing a structure to 
be applied onto a substrate using fer at least one or more cameras, and which m e thod is u se d to 
monitor an application structure or adhesive trail at a_high accuracy and speed while the 
structure d is being applied. 

Therefore an object of 

[0007] It is therefore the obj e ct of the present invention is_to provide a method for 
recognizing a structure to be applied onto a substrate for at least one or more cameras . The ? 
which method facilitates rapid start-up at ajiigh accuracy and permits /e^quick teach-in of the 
reference adhesive trail. 

[0008] Moreover, it is an object of the present invention is to provide a method for 
recognizing a structure to be applied onto a substrate usmgFer at least one or more cameras. 
The , which method monitors an application structure a&d/or adhesive trail at ajiigh accuracy and 
speed while the structure d is being applied. 

[0009] Moreover, it4s-an object of the present invention is_to provide a suitable apparatus 
for carrying out the method according to the invention. 

[0010] In Th e s e objects ar e m e t with regard to the method by the features of claims 1 and 
4, and with r e gard to th e apparatus by th e f e atur e s of claim 28. 

[0010] In th e apparatus according to the invention and the m e thod according to th e 
inv e ntion , a reference application structure is taught-in . In addition, -aftd-further applied 
application structures an4/or adhesive trails are compared to the reference application structure 
for assessment . The th e r e of. In this context, it is irrelevant wh e th e r th e application facility 
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according to the inventionj^M.e.^ the application facility with cameras^ or 5 as a kinematic 

4 

inversion ) can be moved otv the substrate can bei s-being moved. 

[0011] The method according to th e inv e ntion facilitates^ by means of teaching-in a 
reference application structure, and fee-recording and/e? storing of the images of all cameras in a 
sequence of images . The cameras take aft e r by means of just a single scan of said reference 
application structure. The single scan ^his-reduces the time of the unttf start-up of an apparatus 
because of of thi s ty p e to a short teach-in time. 

[0012] According to a method o faccording to the invention for recognizing a structure to 
be applied onto a substrate, (e.g., preferably an adhesive line or adhesive traill-svith-at least one 
camera, (e.g., in particular multiple cameras ), is used , wh e r e by th e appli e d structur e , during the 
scanning for assessment of thethis structure and —is processed in the form of an optical 
representation . Each such that e ach camera records jt*st-a strip of the image to form a sequence 
of images^ and the image recording rate is increased in line with the data reduction achieved by 
recording only a strip of the image . An advantage is that the method , is obtain e d on 
advantag e ous recognition m e thod and monitoring m e thod for a s tructur e to b e appli e d that is 
highly accurate and can be fully automated.- This allows the adhesive trail to be monitored in 
parallel and/or simultaneous to the application of the adhesive trail. 

[0013] Furthermore, an advantage is Furth e r advantag e ous d e v e lopm e nts ar e evid e nt from 
th e subclaims. Accordingly, it is advantag e ous for each camera to record only a strip of the 
image to form a part of atbe-sequence of images in order to minimize the data to be included in a 
the-calculation. Because Sinee less data is has to b e included in the calculation and because of the 
high image recording rate, recording it is f e asibl e to record comparably small partial sections of 
the whole adhesive trail to be recorded (e.g., , for example between 1 mm and 3 mm ) is feasible. 
The , such that th e adhesive trail being a type of vector chain ? can be followed automatically in 
the individual images. 
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[0014] Moreover, another advantage is it is advantageous to increase the image recording 
rate in line with the data reduction achieved by recording only a strip of the image. In , which is 
facilitated in particular by using the use of the partial scanning technique, s uch that a higher 
frame rate is attained. Thus, capturing It is therefor e f e asibl e to captur e images synchronously 
and in parallel by all cameras at the same time is feasible. - The image recording therefore 
proceeds at defined fixed time intervals and is independent of tke-robot control. 

[0015] According to the invention, the image strips of the individual cameras are joined 
to form a single image . Thus, which is advantag e ous in that the respective images of the 
individual cameras are assigned accordingly independent of location and can be recorded and 
processed synchronously In addition, this prev e nt s errors are prevented in the teach-in 
becauses iflee the images of the individual cameras are always assigned with ^correct time and a 
location. 

[00 1 6] According to a preferred embodiment, pee-each camera uses only approx imately a 
quarter ^4/4 of the image lines are used as image strips, thus quadruplin g strip and this is us e d to 
quadruple the image recording rate. Moreover, an advantage of the it is advantag e ous if th e 
parameterization of the sequence of images obtained from the reference application structure is 
that the results from a single image recording run of all cameras is carri e d out automatically 
performed by means of a one-time external indication and/or external marking (e.g. for exampl e 
by means of a mouse click) of the reference application structure . The external marking is r-and 
used for comparison to an applied adhesive trail - In particular, the robot travel path, the robot 
travel time, the direction, the width and the quality of the adhesive trail ftaU are used for 
parameterization - This simple parameterization procedure reduces the risk of maloperation or 
incorrect parameterization . The , whereby the system can be also be operated by personnel with 
low qualifications, qualification also. Thus, thethis parameterization generates a vectorization of 
the entire adhesive trail . The , wher e by this vectorization provides for reliable and automatic 
monitoring of the adhesive trail because of the high image recording rate.- In addition, the 
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vectorization this results in a switch of the sensor head aely-requiring recalibration a«d/or new 
calibration without having to teach-in th e adh e sive trail again the adhesive trail , 

[0017] According to a preferred embodiment of the invention, an assessment function, in 
particular a fuzzy assessment is used to analyze the adhesive agent track and/or adhesive trail. 
In ! wh e r e by in particular the width of the pair of edges comprising the right and the left edge of 
the adhesive trail, the mean gray scale value of the projected gray scale value profile between the 
pair of edges, the edge contrast, and the progression of position are included in the calculation by 
means of the assessment function . As , wh e reby, as a result, the adhesive trail can be described 
accurately such that the adhesive trail can be recognized automatically in a reliable fashion. 

[0018] Moreover, an advantage it is advantageous for the edge of the adhesive trail to be 
determined on a surrounding track and/or orbit (e.g. and, in particular , an essentially circular 
track)_-aftd/or circular line in order to capture any progression of the adhesive trail around the 
application facility in a defined area. The ln thi s context, the adhesive trail progresses within the 
surrounding track , which 4ha£-can be elliptical, polygonal^ or approximately circular. 

[0019] According to a preferred embodiment, the center of the circular line or e£-the 
surrounding track essentially coincides with the site from which the adhesive emanates to form 
the adhesive trail . Each , wh e r e by e ach camera monitors at least one segment of the circle 
formed by the circular line. 

[0020] Errors at the transition from one camera to the next can be reduced by having each 
camera monitor at least one overlapping area jointly with at least one adjacent camera. 

[0021] It is particularly advantageous for the angle values of the circular line ranging 
from 0 to 360° te-form a global coordinate system of the individual cameras . A , wh e r e by a 
segment of the circular line is assigned to the images of the individual cameras - As a result, the 
progression of the adhesive trail can be followed by at least one active camera, whereby 
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statements concerning the entire adhesive trail as well as theits position and/or progression of the 
adhesive trail can be made by relatively simple means. 

[0022] According to a preferred embodiment, a first camera covers a range of angles 
ranging from -10° to 130°, a second camera covers a range of angles from 1 10° to 250°, and a 
third camera covers a range of angles from 230° to 10°, where in-ease three cameras are being 
used. 

[0023] Moreover, during the progression of the adhesive trail, an advantage d is 
advantag e ous to switeh-automatically switch from one camera to the next when the adhesive trail 
progresses from the segment of a circular line of one camera via the overlapping area to the 
segment of a circular line of a different camera - As a result, it is feasible to reliably follow the 
progression of the track and/or position of the track, and these are predictable accordingly. 
Therefore, fully automatic switching between neighboring cameras can occur such that the 
parameterization times are reduced. 

[0024] Moreover, an advantage of light emitting diode f it i s advantageous if LED) 
illumination means is the color that of which provides a suitable contrast to the color of the 
application structure isare used for illumination. A In this cont e xt, the color of the light is 
selected such that, according to the principle of complementary colors, a_the-maximal contrast 
between the substrate and the adhesive trail results. An advantage of the invention is fe4s 
advantageous to use infrared LEDs or ultraviolet LEDs (T JV LEDs ) that . . Particular advantag e s 
r e sult from using light e mitting diodes whose particular design allows them to emit red, green} 
and blue light, in particular upon the use of red-green-blue LEDs ( RGB LEDs). t As a 
consequence of using LEDs, the sensor design can be switched to a corresponding adhesive color 
without further reconfiguration. 

[0025] If LEDs with th e triple colored LEDs (e.g. eelers, red, green, and blue} ? are used, 
the most suitable mix e d color combined can be generated for optimal contrast. 
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[0026] According to a further development of the invention, the LEDs are flashed, 
wh e reby, in particular, by utilizing pulses of current (e.g., ranging from e£-1.0 to 0.01 ms)-are 
applied to the diodes. Flashing the diodes, obtains in ord e r to obtain focussed images^-in 
particular of the adhesive trail, while the sensor scans over the substrate. 

[0027] Moreover, for a-three-dimensional positional correction for the application facility 
is performed by means of the stereometry procedure with regard to positional tolerances of the 
individual components and/or tolerances of the j oining seams and provides an advantage ^4tie 
advantageous for the reference contour and/or a feature to be determined by at least two cameras^ 
in order to carry out a three dimensional positional correction for th e application facility by 
means of th e st e r e ometry procedure . 

[0028] Another advantage is lt is also advantageous if the two cameras record the 
substrate, a section of the component, or one or more components in the form of a full image or 
large image, whereby the full images or large images of the two cameras comprise an 
overlapping area in ajeading direction . The , and wh e reby th e three-dimensional recognition of 
the reference contour position results r e sulting in the overlapping area is— used for gross 
adjustment of the application facility prior to applying the structure. Thus ln this cont e xt , 
corresponding correction values are transmitted to the application facility and/or the robot in 
order to shift the application facility's or robofs its coordinate system for the application of the 
adhesive agent. 

[0029] If a projection is made onto the area of the reference contour for a three- 
dimensional analysis, (e.g., in particular if one or more laser lines are applied to the substrate in 
the form of a projection), then the a -three-dimensional analysis of the profile with regard to the 
height and contour of arbitrary components can be facilitated even though this is not analyzable 
by common image processing without an additional projection. 

[0030] An advantage of the invention is Advantag e ouslv, the individual cameras are 
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calibrated in order to assign the angle assignment of the individual cameras according to athe 
circular caliper . In , wh e r e by in p articular a circular arc of the calibrating device with marker 
points at 0°, 120°, and 240° for three cameras is used. The marker points allo w This allows a 
global coordinate system to be used with regard to the angle assignment for the individual 
cameras on the circular caliper around the application facility in order to simplify the processing 
by the-software. 

[0031] Moreover, the distance of the application facility from a feature of the component 
(e.g., , for e xampl e from edges of the component* or holes and/or breakthroughs)^ can be 
measured in order to carry out a positional test of the applied structure. A In thi s context, in 
particular, a line-shaped gray scale value scan andA)r line-shaped calipers are used for distance 
measurement such that any number of reference markers can be placed along the circular arc of 
the calibrating device. Image , wh e reby th e imag e processing is not exclusively limited to the 
camera image, in which the adhesive trail test is carried out.- It is therefore not necessary for the 
adhesive trail and suitable reference features to be visualized in the same camera image, which 
has a is of advantag e in particular advantage with regard to the parallel processing of three 
camera images. 

[0032] The present invention also p rovides an apparatus for recognizing a structure to be 
applied onto a substrate, preferably an adhesive line or adhesive trail, for carrying out a m e thod 
according to the inv e ntion according to claim 1 and/or 4, whereby at least one illumination 
module and one sensor unit are provided.- The sensor unit includes is made up of at least one or 
more cameras that are provid e d and arranged around the facility for applying the structure . Each 
camera is . and each are directed at the facility for applying the structure.- By this means i^4s 
feasibl e for the travel path of the facility over a substrate and/or a travel path of the substrate 
relative to the application facility to be monitored at all times in all directions by means of 
directing the cameras at the application facility is feasible . 
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[0033] An advantage is iff f the axial longitudinal axis of the individual cameras 
essentially intersects (e.g. , in the direction of vie w or along , the axial longitudinal axis of the 
application facility)^ , it is advantageous according to a developm e nt of this type that a narrow 
area around the application facility can be monitored at suitable resolution and high image 
recording rate. 

[0034] According to a preferred embodiment, individual cameras, in particular 3 
cameras, are arranged at equal distances from each other in the direction of the circumference. 

[0035] The Advantag e ously, th e individual cameras are circuited such that the images of 
the cameras are stored in a sequence of images such that these images can be recorded 
synchronously and in parallel as well as in an assigned fashion. 

[0036] According to a development of an invention, one or more cameras form a circular 
caliper whose center is formed by the application facility of the structure. Thus ln this cont e xt , 
one or more circular calipers can be used to tha^facilitate the determination of the edge of the 
adhesive trail on a circular line. 

[0037] According to a preferred embodiment, the individual cameras comprise an 
overlapping area of at least 1 0° each relative to the adjacentn ext camera.- This overlapping area 
facilitates fully automatic switching between neighboring cameras when the adhesive trail 
progresses from the monitoring area of one camera to the next . Because r-sinee the selection of 
the camera is not bound to the robot position or a time component, but rather always refers to the 
actual inspection results . For instance, the insepection results are , i. e . is based on the 
arrangemen^-en the orbit , the and/or circular line of the circular caliper^-andA)r the global 
coordinate system formed thereby. 

[0038] Moreover, an advantage is it i s advantageous for the illumination module to 
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comprise light emitting diodes ( be made up of LEDs), in particular infrared LEDs, ultra-violet 
LEDs ( UV LEDs) or Red-Green-Blue LEDs ( RGB LEDs). 

[0039] Moreover, an it is of advantage is the te-use of a calibrating disc with individual 
form elements for calibrating the individual cameras for the assignment of the angle assignment 
The , whereby said form elements comprise, in particular, an angle distance of essentially 10°^ 
which 0 . This allows for assignment of a_the-scaling factor, an angle assignment, and acenter and 
a as well as radius of the search circle for the individual cameras - According to the invention, 
the calibrating disc comprises at least three marker sites that are arranged in a circular arc of the 
calibrating disc (e.g. of-essentially located at 0°, 120 °, and 240 °), in order to calibrate the three 
cameras. 

[0040] If a projection facility projecting one or more features (e.g. , in particular strips)^ 
onto the substrate for the three-dimensional analysis is provided on the application facility, 
arbitrary components can be used for correction and/or adjustment of the application facility 
prior to applying the structure. 

[0041] According to a preferred embodiment, the projection facility emits one or more 
laser lines for a_three-dimensional profile analysis - Arranging at least two projection facilities 
around the application facility facilitates a_gap-free three-dimensional analysis around the 
application facility . Means of image processing , whereby the analyses of a_sealing agent height 
and ensealing agent contour^ as well as_a position and a^width^ can be performed carried out 
according to the principle of triangulation by m e ans of image proce ssi ng . 

[00 4 3] Advantages Furth e r advantag e ous d e velopm e nts of th e inv e ntion ar e th e subj e ct of 
the remaining subclaims. 

[0042] Advantag e ous d e v e lopm e nts of the invention shall be illustrated in an exemplary 
fashion by means of the following drawings. 



10 



Atty. Dkt. No.: QU01H10/P-WO2/US (22589-14US) 

PATENT 

Express Mail No.: EB 270750868 US 

SUBSTITUTE SPECIFICATION MARKED-UP 

[0043] Figure 1 illustratess hews a schematic side view of an apparatus according to the 
invention for application and monitoring of an adhesive trail. 

[0044] Figure 2 illustratess hews a perspective view of an the-apparatus according to the 
invention of Figure 1 . 

[0045] Figure 3 illustrates a shows th e travel path of the apparatus according to the 
invention for application and monitoring of an adhesive trail. 

[0046] Figure 4 illustrates shews another travel path of the apparatus according to the 
invention with regard to the-switching of athe relevant camera. 

[0047] Figure 5 is a view of a single image composed of three image strips from three 
cameras for online monitoring. 

[0048] Figure 6 illustrates the principles of the design of the software. 

[0049] Figure 7 illustratess hews a schematic view of a calibrating device according to the 
invention for calibrating the-individual cameras of an the-apparatus according to the invention for 
recognizing a structure to be applied onto a substrate. 

[0050] Figure 8 illustratess hews a top view onto ajhe-substrate with an the-adhesive trail 
applied with regard to monitoring the application. 

[0051] Figure 9 illustrates shews a top view with regard to analysis of the profile. 

[005 4 ] In the following; th e d e sign of the apparatu s according to the invention for 
r e cognizing a s tructure to be appli e d onto a substrat e i s illustrat e d according to Figures 1 and 2. 

[0052] Reference number 10 illustrated in Figure 1 indicates the-schematically an shown 
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apparatus for the application and recognition of an adhesive trail 20. T In the center of the 
apparatus according to th e inv e ntion is arranged an application facility 1 1 by means of which an 
adhesive trail 20 is applied onto a substrate 30 and/or onto a sheet of metal 31 3Q-proceeding 
from right to left in Figure 1. JThree cameras 12, 13, 14 are arranged at equal distances from 
each other in a circle around the application facility IK wherein ifr each camera of which is 
directed at the application facility 1 1.- As is evident from Figure 1, the axial longitudinal axes of 
the three cameras 12, 13, 14 intersect the axial longitudinal axis of the application facility 1 1 just 
below the substrate 30 such that the focus of the individual cameras 12, 13, and 14 are i s 
arranged righ^around the area of the application facility 1 1, in particular, on a circular line. 

[0053] In the inspection of the adhesive, either the application facility 11 with the 
cameras 12, 13, and 14 or the substrate 30Js moved . The 5 wh e r e by th e adhesive trail 20 is 
simultaneously applied to the substrate 30 by means of the application facility 11, and-whereby 
the cameras 12, 13, 14 monitor the applied structure. Thus, For this purpos e , it is f e asibl e to 
mov e either the application facility 11 with the cameras 12, 13 and 14 or the substrate 30 can be 
moved, info order to apply the adhesive trail 20 onto the substrate 30 such as to follow a desired 
progression.- By this means, the moving cameras 12, 13, and 14 being moved along can monitor, 
independent of the path of travel, the adhesive trail 20 ftaU at the time the adhesive trail 20 rt is 
being applied.- In Figure 2, the adhesive trail 20 progresses from left to right and is indicated by 
a continuous line. _The intended progression of the adhesive trail 20 is indicated by a dashed line 
to the right of the application facility 1 1 . 

[0054] Figure 3 illustrates th e n shows the progression of the adhesive trail 20 as indicated 
by arrows 21 and 22 , whereby the direction and/or field of view of the three individual cameras 
12, 13 and 14 are is shown in three sites 23, 24 and 25. t The field of view 23, 24, and 25 of the 
three individual cameras 12, 13 and H eaeh is indicated by a rectangle drawn with a continuous 
line24, a rectangle drawn with widely dashed lines 23, and a rectangle drawn with narrow dashed 
lines 25. t As shown in i s e vident from Figure 3, the direction of the individual fields of view 23^ 
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24, and 25 of the cameras 12, 13 and 14 remains constant at all times whereby only the whole 
apparatus is moved along the adhesive trail 20 as indicated by arrows 21 and 22 . 

[0055] Figure 4 illustrates shows another progression of an adhesive trail 20, whereby a 
field it is indicat e d in e ach cas e , which h e ld of view is active G4re. g., shown as "aktiv") 27, The 
which camera 12, 13 and 14 having the corresponding active field of view is_shown as an "aktiv" 
a-rectangle 27 is active while traveling along the adhesive trail 20. 

[0056] Figure 5 illustrates then show s three image strips 32, 33, and 34 that which each 
represent a relevant section att4/or strip of anjmage of the three individual cameras 12. 13 and 
14 (shown in e £-Figure 1 ). Each camera!2. 13 and 14 . According to the m e thod of tho 
inv e ntion, e ach cam e ra records jus^a strip of the image in order to reduce the amount of data 
accordingly such that the recording rate can be increased - These individual image strips 32, 33, 
and 34 of the three cameras 12, 13 and 14 are then joined into an image . The , wh e r e by the 
image recording occurs at defined fixed time intervals and ^independent of the robot control of 
the application facility 11. t For example, the cameras 12, 13 and 14 only record a strip of the 
image having an image height of approximately 100 pixels (e.g., 100 image lines is used) ? 
wher e by instead of an image height of 450 pixels^ an imag e h e ight of approx. 100 pixels (100 
imag e lines) is us e d. By means of this partial scanning technique Jj-i.e.^ partial reading-out of the 
image recording chip) T only small data streams are generated such that the image recording rate 
can be increased several-fold . Therefore, accordingly. It is ther e for e f e asible in the data analysis 
simultaneously captures to captur e the three image strips that are arranged one below the 
otherimages of the individual cameras 12, 13 and 14 synchronously and in parallel and th e refore 
simultaneously and then joins j om-them into a single image ^, wher e by th e thr ee Image strips ar e 
arrang e d one b e low the other. As a result, the three imag e s, i. e . th e thr ee image strips, are 
correctly arranged and assigned with regard to allocation and a time Ume relative to each other 
and without delay further ado, and can be processed accordingly. The This-specific image 
recording technique th e refor e facilitates simultaneous and parallel recording of individual 
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camera 12, 13 and 14 that are stored as a sequence of images. Thus, the structure can be 
scanned imag e s, whereby it becomes feasibl e to s can this structure just once during the teach-in 
of a reference application structure , wh e reby th e images of all cameras ar e stor e d in a sequ e nc e 
of images . 

[0057] Once the images of the three cameras 12, 13 and 14 are stored in a sequence of 
images, a parameterization of the t his-reference track is carried out as the subsequent step of 
teaching-in the reference adhesive trail. A The-robot travel path, arobot travel time, ^direction, 
a width, and a quality of the adhesive trail 20 are utilized to determine us e d in the 
parameterization. A This results in a type of vector chain for the adhesive trail 20 results, which 
allows a jo attain the high image recording rate to be attained via a nd-comparablv short partial 
sections (e.g., between 1 and 3 mm).- Vectorization has another advantage, in that* the adhesive 
trail 20, being in the form of a vector chain, can be stored in a camera-transcending global 
coordinate system. 

[0058] As shown in is e vid e nt from the bottom strip 34 of Figure 5, a circular line is 
arranged around the center of the application facility 11 , and in th e form of a circular lin e , 
wh e r e by th e two edge points 21 and 22 of the adhesive trail 20 are arranged on the circular line. 
The : £hjs-circular line is subdivided into a plurality of angles to provide overlapping areas of the 
individual cameras 12. 13 and 14 to facilitate a gapless coverage around the application facility 
11. A such that a range of angles from -10° to 130° is assigned to a first camer a. A ^-a range of 
angles from 1 10° to 250° is assigned to a second camera, and a range of angles from 230° to -10° 
is assigned to a third camera. A such that gapl e ss cov e rag e around th e application facility 1 1 by 
ov e rlapping areas of th e individual cam e ras i s facilitat e d. From this re s ults a global 
coordinant coordinat e system results for the three image strips , which -tha^can be provided as 
either t o-be-Cartesian coordinants or pola r coordinants . 

[0059] If the adhesive trail 20 progresses out of the field of view 23, 24 and 25 of a 
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camera, the adhesive trail 20 frail is transiently in the overlapping area of the ranges of angles of 
the adjacent two cameras - If the adhesive trail 20 then progresses from the segment of the 
circular line of the one camera via the overlapping area to the segment of the circular line of 
another camera, an automatic switch is made from the one camera to the other camera - This is 
illustrated shown, in particular, in Figure 4 by means of the active fields of view 27 of the 
individual cameras 12, 13 and 14 . 

[0060] The advantages m e ntioned above ar e attained by the individual cameras 12, 13 
and 14 form forming a circular caliper whose center is formed by the application facility 1 1 , 
whereby the search for the edges 21, 22 of the adhesive trail 20 p roceeds on a circular line 
directly around the application facility 11. The . For this purpos e , it is e ss e ntial that the 
individual cameras 12, 13 and 14 are essentially directed at the application facility 1 1 such that ; 
wh e r e by t he axial longitudinal axes of the individual cameras 12, 13 and 14 intersect the 
longitudinal axis of the application facility 1 1 . 

[0061] A teach-in run and/or a teach-in of a reference adhesive trail is 
described i 1 1 u s trat e d in the following paragraphs . 

[0062] The teach-in process of the reference adhesive trail can be started by a_&e~user by 
means of a mousey-click by selecting a on th e track that which indicates the position of the 
adhesive trail 20. t This is sufficient for fully automatic recognition of position and direction of 
the adhesive trail 20jn the subsequent camera images because? -sinee the image recording rate is 
sufficiently high and the individual images are recorded very shortly after one anothe r (e.g.^r ^er 
example every 1 mm to 3 mm]L_T From the starting point, the adhesive is scanned image by 
image, whereby the adhesive trail position and the adhesive trail angle detected in the current 
image are used as a priori knowledge for the upcoming image^ as a priori knowl e dg e . The fact 
that the track radii usually exceed 20mm facilitates fully automatic capture of the adhesive trail 
20 without a human being-having to determin e and/or assess the image and/or the position of the 
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adhesive trail 20. t As a result, the search area can be limited which allows, by means of the high 
image recording rate, a determination of to calculat e where the adhesive trail 20fr ail will 
essentially progress tojn the following image. 

[0063] Figure 6 [NEEDS TO BE TRANSLATED FOR IS IN GERMAN] 
illustrates shews the principles of the design of the software, whereby the teach-in run and/or 
teach-in run generates the image sequence . The image sequence, , which in turn, facilitates the 
automatic parameterization.- This parameterization can be prepro-set by the user, if applicable, 
and is used togetheri oH#y with a progression file for inspection of an applied adhesive trail 20 
during the inspection run. 

[0064] The online monitoring of an applied adhesive trail 20 is described shall b e 
illustrat e d bri e fly in the following paragraphs, t The application facility 1 1 (shown in Figure 1 } 
applies the adhesive trail 20 onto the metal sheet 31^0, whereby the application facility J_Hi is 
moved togetheri ein#y with the-cameras 12, 13 and 14 over the metal sheet 31. 30r However, a 
kinematic inversion lnversion is also feasible, for instanced the metal sheet 31 is 30 b e ing 
moved and the application facility 1 L including wkh the cameras 12, 13 and 14, is bemg arranged 
in a to be fixed itt-position- The applied adhesive trail 20 is determined and analyzed 
synchronously and in parallel by the cameras 12, 13, 14 . The cameras 12, 13, and 14 are on the 
circular line of the circular caliper (shown in illustrat e d according to Figure 5), whereby each 
camera 12, 13, and 14 records only a strip of the image . The strips of images are joined -and 
joins thes e into a single image that t e~form a sequence of images. Thus, the The-image recording 
rate is increased in accordance with the data reduction attained by each camera 12, 13 and 14 
recording only a strip of the image . The , wh e r e by th e individual image strips in the joint image 
facilitate the synchronous and parallel as well as simultaneous capture of the three camera 
images . The , and wh e r e by the individual images of the three cameras 12, 13 and 14 can be 
assigned directly as a function of location.- As a result, online monitoring of the adhesive trail 20 
is feasible in real-time and is feasibl e that achieves high accuracy at high travel speeds due to the 
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high image recording rate both in teaching-in a reference adhesive trail and in the inspection of 
the applied adhesive trail 20. The . In this context, the information concerning the adhesive trail 
20 in the adhesive search area, the angle assignment of the sensor, the adhesive assessment, the 
robot travel path, and the robot travel time are summarized in a progression list. 

[0065] According to an embodiment of the present invention, an assessment function 
(e..g , in particular a fuzzy assessment^ can be used to find the edges of the adhesive trail 20. 
Mn order to determine and assess the adhesive trail_20, the following parameters described 
below are included in the calculation of a fuzzy assessment. 

[0066] A width of a Width of th e pair of edges (e .g., an e dge 1 : _a_left edge of the 
adhesive trail ; an? edge 2: a right edge of the adhesive trail), a_mean gray scale value of the 
projected gray scale value profile between the pair of edges, an edge contrast ( e.g., a geometric 
mean of the-amplitudes of the two edges), and ^progression of position ( e.g., directed deviation 
of athe-center between thetie two adhesive edges from ajhe-center of the search area, in pixels). 
By means of this plurality of parameters^ and the use of the fuzzy assessment function, the 
adhesive trail 20 can be describ e d v e ry accurately such that the adhesiv e frail can be r e cogniz e d 
automatically recognized in a reliable fashion and described very accurately . 

[0067] The illumination module (not shown ) for h e r e ) far the apparatus according to the 
invention comprises light emitting diodes ( is mad e up of LEDs), in particular infrared LEDs, 
ultra-violet LEDs ( UV LEDs) or Red-Green-Blue LEDs ( RGB LEDs^t In order to attain ahigh 
contrast in anjmage recording, the LEDs can be flashed . For instance^ -her short, strong pulses of 
current on the order of 1.0 to 0.01 ms can be applied to the diodes. Thus, the ln this context, 
light-emitting diodes are capable of emitting light of various colors . The advantage is- are 
particularly advantag e ous such that ajhe-sensor design can be switched to other types of 
adhesives or adh es iv e and/or colors of adhesives without reconfiguration. 

[0068] Figure 7 illustrates shews a calibration facility 40 in the form of a circular 
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calibrating disc . The calibration facility 40 assigns in order to a ss ign to the individual cameras 
12, 13 and 14 a thetr-scaling factor, anthei? angle assignment, and a&e-center as well as ajhe 
radius of ajhe-search circle - The calibrating disc consists of individual form elements 41 (e.g. 
shown as and/or dots) 4 1 t hat are arranged on a circular line and-at an angle distance of 
approximately 10° e ss e ntially 100. Moreover, marker sites 42 are arranged at an equal distance 
from each other in order to calibrate three cameras 12, 13 and 14. r A compensation calculation is 
used to calculate from the coordinates of the centers of the individual dots, on the one hand, the 
scaling factors of the individual cameras 12, 13 and 14 and, on the other hand, the center as well 
as the radius of the search area.- The marker sites at angles of 0°, 120°, and 240° in the global 
coordinate system allow the angle assignment and the corresponding fields of view of the 
individual cameras 12, 13 and 14 to be determined - The field of view of the individual cameras 
12, 13 and 14 is indicated, in particular, by the three rectangles 43, 44, and 45 (shown in Figure 
7 ). The , wh e r e by th e form elements 41 can correspond to the circular line of the circular caliper 
for detection of the adhesive trail 20. 

[0069] Figure 8 illustrates skews the application facility 1 1, whereby the strips 31, 32, and 
33 around the application facility 1 1 each are shown by dashed lines that a ad-represent the read- 
out area of the individual cameras 12, 13 and 14. Two active cameras monitor the adhesive trail 
20, such that the T -^Fhe-adhesive trail 20 is monitored in the overlapping area of the strips 32 and 
33 ^ such that th e se two camera s are activ e . If the progression of the adhesive trail 20 relative to 
the application facility 1 1 changes, only one , if applicabl e , of the cameras 12, 13 and 14 becomes 
active, and wh e r e by an essentially circular caliper (not shown) that is arranged to be concentric 
around the application facility 1 1 is used for this purpos e. 

[0070] According to this embodiment, the circular caliper is formed by multiple cameras 
12, 13 and 14 that are arranged around the application facility 11, but, in particular, can be 
attached at a different radius from the center of the application facility 1 1 . For an inspection of 
an application structure and/or adhesive trail_20, the cameras 12, 13 and 14 are directed at a 
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circle and/or circular line whose center coincides with the center of the application facility 11. 
The optical detection of the adhesive trail 20 as illustrat e d above then proceeds on this circular 
line. 

[0071] Figure 9 illustrates a The-three-dimensional profile analysis by means of a 
projection is d e scrib e d according to Figur e 9, in order to provide for a positional correction of 
the application facility 1 1 . t For reasons of clarity of presentation, FiRurefi gr 9 illustrates agaffl 
shows only two camera fields of view 51, 52 indicated by the dashed lines - In the overlapping 
area of the two camera fieldsheWs of view 51, 52 are shown a plurality of laser lines 60 that are 
used for profile analysis with regard to the width and contour of structure lines , the laser lines 
are also used - and-for generation of so- call e d s o ft contours.^ The laser lines 60 are generated by a 
projection facility that can, for example, be arranged on the optical sensor with three cameras 12, 
13 and 14. t Moreover, the projection facility can also i ust as w e ll be arranged directly on the 
application facility 1 1 .- The sensor with the three cameras 12, 13 and 14 is shown schematically 
by the circle 70.- The laser lines 60 and/or laser strips projected onto the component 30 and/or 
metal sheet 3130 highlight contours on the component that cannot be used for three-dimensional 
analysis by conventional image processing - Artificial features are generated by means of the 
laser lines 60 tmes on the component and can subsequently be analyzed by means of image 
processing utilizing according to stereometry.- Thus, Figure 9 illustrates shews the principle of 
three-dimensional positional recognition prior to the application of a_sealing agent wherein-ease 
no hard, analyzable features are present. 

[0075] Th e r e for e , a m e thod and/or an apparatus for r e cognizing a structur e to b e applied 
onto a substrat e , pr e ferably an adh e siv e line or adhe s iv e trail, with at least one camera, in 
particular multiple cam e ras, is d e scribed. In this cont e xt, th e teach in of a ref e r e nc e application 
structure is carri e d out by m e ans of just a singl e scan of s aid refer e nce application structur e such 
that the images of all cameras ar e stored in a sequ e nce of imag es . 
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[0076] Mor e ov e r, a m e thod for recognizing a structur e to b e appli e d onto a substrate i s 
d e scrib e d, wh e reby th e applied structur e is proc es s e d as an optical repre s entation during th e scan 
for ass e ssm e nt of th e structur e such that each camera r e cords just a strip of th e imag e to form a 
s e quenc e of images and the imag e recording rate is increas e d in line with the data r e duction 
achi e v e d by r e cording only a strip of th e imag e . 
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METHOD AND APPARATUS 
FOR RECOGNIZING A STRUCTURE 
TO BE APPLIED ONTO A SUBSTRATE 
WITH MULTIPLE CAMERAS 



ABSTRACT OF THE DISCLOSURE 

A method and apparatus are provided for automatic application and monitoring of a 
structure to be applied onto substrate. A plurality of cameras positioned around an application 
facility are utilized to monitor the automatic application of a structure on a substrate by means of 
a stereometry procedure. Three-dimensional recognition of a reference contour position results 
in the overlapping area to be used for gross adjustment of the application facility prior to 
applying the structure. 
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